Energy costs for data centers continue to rise, already exceeding $15 billion yearly. Sadly, much of this power is wasted. For example, many networked services, such as the FACEBOOK® social-networking service and AMAZON.COM® e-commerce service, are provided by multi-tier data center infrastructures. A primary goal for these applications is to provide good response time to users; these response time targets typically translate to some response time service level agreements (SLAs). In an effort to meet these SLAs, data center operators typically over-provision the number of servers to meet their estimate of peak load. These servers are left “always on,” leading to only 10-30% server utilization, despite virtualization. This is problematic, because servers that are on, but idle, still utilize 60% or more of peak power.
To reduce this waste, various researchers have considered intelligent dynamic capacity management, which aims to match the number of active servers with the current load. A goal of dynamic capacity management is to scale capacity with unpredictably changing load in the face of high setup costs. Part of what makes dynamic capacity management difficult is the setup cost of getting servers back on/ready. All of the prior work in this area of which the present inventors are aware has focused only on fluctuations in request rate. This is already a difficult problem, given high setup costs, and has resulted in many policies, including reactive approaches that aim to react to the current request rate, predictive approaches that aim to predict the future request rate, and mixed reactive-predictive approaches. However, in reality there are many other ways in which load can change. For example, request size (work associated with each request) can change if new features or security checks are added to the application. As a second example, server efficiency can change, if any abnormalities, such as internal service disruptions, slow networks, or maintenance cycles, occur in the system. These other types of load fluctuations are all too common in data centers, and have not been addressed by prior work in dynamic capacity management.